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(54) Deblurring image data using motion vector dependent deconvolution 

(57) The deblurring deconvolution is dependent upon the capture time T of the image and the motion vector 
detected for that part of the image. A multiplexer 14 selects between deblurring deconvolutions performed in 
parallel in dependence upon a plurality of different candidate motion vectors VI, V2, V3 and zero motion. A 
region processor 10 identifies boundaries between areas within the image corresponding to different objects 
and controls a mixer 16 to selectively output deblurred values or unprocessed raw video values so as to 
reduce edge effects due to the inappropriate application of deblurring deconvolution at the edges between 
objects. Provision of parallel deconvolutions allows the system to operate in a continuous pipelined manner. 
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IMAGE ENHANCEMENT 



This invention relates to the field of image enhancement. More 
particularly, this invention relates to the enhancement of arrays of 
5 image data captured during a capture time such that moving images are 

blurred . 

The blurring of moving images often occurs due to the non-zero 
capture times of image capture devices, such as stills cameras and 
television cameras. Blurring can be produced by decreasing the capture 

10 time, but this brings with it its own set of problems, such as the 

requirement for higher light sensitivity. The problems of image 
blurring are made worse when the scene contains rapidly moving images 
(e.g. motor racing) or the capture device itself is moving relative to 
the image (e.g. satellite photography where long capture times are also 

15 required) . 

It is known that blurring due to image motion can be reduced 
using deconvolution techniques. The image data value at a given point 
within the blurred image may be considered as a convolution over the 
capture time of the actual instantaneous image at that point and the 

20 motion. This convolution can be reversed given values of the variables 

representing the capture time and the motion of the image assuming the 
motion can be approximated to uniform linear motion. 

A mathematical treatment of this technique may be found in 
"Digital Image Processing" by Gonzalez & Wintz published by Addison 

25 Wesley 19&7, pages 224-232. The convolution in the spatial domain can 

be considered as multiplication by a transfer function in the spatial 
frequency domain, the transfer function being capable of solution in 
terms of the capture time and speed of movement of the image. 

Such techniques have been used to enhance images , such as still 

30 images and satellite images, where the degree of motion present is 

either known from the outset (e.g. a fixed satellite orbit) or may be 
determined by trial and error attempts at deblurring. Whilst the 
deblurring technique can be effective, the problems of solving the 
transfer function have meant its application has been limited to 

35 specialist fields and it may be considered as something of a textbook 

curiosity. 

Viewed from one aspect this invention provides apparatus for 



2 



processing an array of image data captured during a capture time such 
that moving images are blurred, said apparatus comprising: 

means for detecting an array of motion vectors associated with 
corresponding portions of an array of image data, each of said motion 
5 vectors representing image motion of a corresponding one of said 

portions between temporally spaced arrays of image data; and 

means for deconvolving a blur function from each of said portions 
in dependence upon said capture time and a corresponding motion vector 
for that portion detected by said means for detecting to yield an array 
10 of reduced blur image data. 

The invention both recognises and exploits that the systems that 
have been developed for motion compensated interpolation of image data 
to effect standards conversion can be utilised to detect motion vectors 
for moving images as part of an image deblurring system of general 
15 applicability. Furthermore, differing motion vectors can be identified 

for differing parts of a complex moving scene and the appropriate 
deblurring convolution applied to the corresponding parts of the scene - 
This considerably enhances the effectiveness of the technique. 

In order to ease the burden of identifying the motion vectors, it 
20 is preferred that said portions comprise arrays of pixel values. 

Such an approximation of associating motion vectors with a block 
of pixel values provides an advantageous compromise between system 
processing requirements and the achieved accuracy of results. 

Having identified motion vectors to be associated with each of 
25 the arrays of pixel values, it is important to provide at least some 

flexibility as to which motion vectors are actually utilised. 
Accordingly, it is preferred that said means for detecting an array of 
motion vectors detects a plurality of candidate motion vectors for each 
of said arrays of pixel values and a means for selecting selects a 
30 motion vector to be associated with each pixel value from among said 

candidate motion vectors . 

The provision of a plurality of candidate motion vectors for each 
of the portions (/arrays of pixel values) from among which a particular 
motion vector to be used is selected on a pixel by pixel basis, 
35 provides for differences between what the pixel values within a given 

portion in fact represent without inflicting unduly disadvantageous 
processing requirements . 



The deconvolution process required to produce a particular output 
pixel value requires contributions from a number of surrounding pixel 
values. In order to improve the efficiency of the system, it is 
desirable that the image data should be input to the system as a 
5 continuous stream that can be passed through the system without undue 

requirements for buffering and the like. This presents a problem in 
the case of deconvolution that may be based upon one of a number of 
candidate motion vectors. In such a circumstance, the particular 
deconvolution that is required may not be identified until after at 

10 least some of the pixel values upon which the result depends have 

passed through at least part of the apparatus. In order to deal with 
this, it is preferred that said means for deconvolving performs a 
deconvolution upon each pixel value for each of said candidate motion 
vectors to yield a plurality of reduced blur pixel values and a 

15 multiplexer controlled by said means for selecting switches one of said 

candidate reduced blur image pixel values corresponding to said 
selected motion vector to form part of said array of reduced blur image 
data. 

The performing of a plurality of deconvolutions when only one 

20 will ultimately be used, may seem at first sight to disadvantageous^ 

increase the amount of hardware that will be needed to implement the 
system. However, when consideration is made that the processing may 
then be made in a continuous pipelined fashion without the need to 
repeatedly access the same pixel data values, then an overall advantage 

25 results from such an approach. 

As mentioned previously, the ability of the system to apply 
different motion vectors to different parts of the image allows a scene 
containing multiple moving images to be dealt with. Such scenes do 
however create problems at the boundaries between objects within the 

30 scene. As a first stage in dealing with such problems it is desirable 

that the system should include means for detecting boundaries between 
areas within said array of image data sharing a substantially common 
motion vector, sharing of a substantially common motion vector within 
such an area being indicative of said area representing a single object 

35 (the object may comprise a plurality of different parts, but they all 

move together). 

Having identified the boundaries between objects, the problems 



that could be introduced by an inappropriate application of a 
deblurring deconvolution at a boundary between different objects can be 
reduced by providing a mixer responsive to said means for detecting 
boundaries for mixing together deblurred image data from said means for 
deconvolving and unprocessed image data such that at boundaries between 
detected areas substantially wholly unprocessed image data is output 
from said mixer with an increasing proportion of deblurred image data 
being output from said mixer upon moving away from said boundaries. 

It is preferably that the mixer should select unprocessed blurred 
image data at boundaries rather than risk the introduction of the 
potentially more disturbing artifacts due to an inappropriate 
deblurring deconvolution being applied. 

A particularly efficient embodiment of the means for deconvolving 
comprises a two dimensional finite impulse response filter having a set 
of filter coefficients selected in dependence upon motion vector 
components in orthogonal directions for that portion of image data 
being processed. 

The realisation that the blurring transfer function operates in 
the spatial frequency domain of the image makes the use of a finite 
impulse response filter for transformation back into the spatial domain 
particularly suitable. Furthermore, such a filter is easy to operate 
in a pipelined fashion whereby the data is sequentially clocked along 
the filter to yield a filtered output result upon each clock cycle. 

It will be appreciated that the filter coefficients for the 
finite impulse response filter is dependent upon the orthogonal motion 
vector components and the capture time of the image. In order to 
efficiently deal with such variations, it is preferred to provide a 
filter coefficient store for said filter storing a set of filter 
coefficients for a range of possible motion vectors and capture times 
that are triggered into use in dependence upon detected motion vector 
components and the particular capture time in question. 

Viewed from another aspect the invention provides a method of 
processing an array of image data captured during a capture time such 
that moving images are blurred, said method comprising the steps of: 

detecting an array of motion vectors associated with 
corresponding portions of an array of image data, each of said motion 
vectors representing image motion of a corresponding one of said 



portions between temporally spaced arrays of image data; and 

deconvolving a blur function from each of said portions in 
dependence upon said capture time and a corresponding detected motion 
vector for that portion to yield an array of reduced blur image data. 

An embodiment of the invention will now be described, by way of 
example only, with reference to the accompanying drawings in which: 

Figure 1 illustrates an apparatus for deblurring image data; and 

Figure 2 illustrates a two dimensional finite impulse response 
filter for effecting image deblurring. 

Figure 1 shows an apparatus for image deblurring. A video data 
stream is input to a vector selector 2 and first, second and third 
deconvolution means 4, 6 and 8 respectively. Four candidate motion 
vectors (VI, V2, V3 and the zero motion vector) for a block/portion of 
the image are input to the vector selector 2. It will be appreciated 
that the candidate motion vectors are detected using apparatus and 
techniques such as those described in British Published Patent 
Application GB-A-2 231 7^9 (Sony Corporation) for the purpose of motion 
compensated standards conversion. 

One of the candidate motion vectors is the zero motion vector. 
The other candidate motion vectors VI, V2 and V3 are fed to the 
respective deconvolution means A , 6 and 8 . 

The vector selector 2 selects from among the candidate vectors 
for the block (Vectors/b) on a pixel by pixel to yield basis that 
vector (Vectors/p) which is most suitable for a particular pixel. In 
accordance with the known technique, this selection may be achieved by 
projection from one area of image pixel values to another using each of 
the vectors and testing to determine how well the pixel values in the 
target matches those source. 

The particular vectors selected for each pixel (Vectors/p) are 
passed to a region processor 10 where an array of vectors selected for 
all of the pixel values is assembled and analysed as will be described 
later. The vector selector 2 also outputs a pixel vector choice value 
to a delay unit 12 through which it passes to a multiplexer 14. 

The multiplexer 14 receives the outputs from the first, second 
and third deconvolution means 4, 6 and 8 respectively. The multiplexer 
14 also receives the unprocessed video data in one channel 
corresponding to the zero motion candidate motion vector. The 



J 



multiplexer 14 selects one of its inputs for output to a mixer 16 in 
dependence upon the output of the delay 12. The delay 12 is chosen to 
match the delays in the vector selector 2 and first, second and third 
deconvolution means 4, 6 and 8. The first, second and third 
5 deconvolution means 4, 6 and 8 each serve to generate deblurred pixel 

values as will be described below. 

The region processor 10 serves to identify boundaries between 
objects within a scene of moving objects. This is achieved by 
searching through the motion vector selected for the various pixel 

10 positions within the image and identifying areas showing substantially 

common motion vectors within a predetermined threshold of one another. 
When these areas of common motion vectors have been identified then 
their boundaries may be traced to represent the boundaries of differing 
moving objects within the scene. In synchronism with the output of the 

15 deblurred pixel value from the multiplexer 14, the region processor 10 

outputs a boundary flag via a delay unit 18 to the mixer 16. This 
boundary flag is a multi-bit word indicating the proximity of the pixel 
now being output to a boundary between objects within the scene. If 
the boundary flag indicates that the particular pixel is at the 

20 boundary between two objects, then the mixer 16 selects the unprocessed 

video for output. As the boundary flag progressively indicates that a 
pixel is further away from a boundary between moving objects, then the 
mixer 16 correspondingly selects a higher proportion of the output from 
the multiplexer 14 representing the deblurred pixels rather than the 

25 raw unprocessed pixels. The action of the mixer 16 and the region 

processor 10 serves to suppress the generation of disturbing artifacts 
around the borders of moving objects at which the deblurring 
deconvolutions would not be appropriately applied. 

Figure 2 illustrates one of the deconvolution means 4, 6 and 8 in 

30 more detail. The deconvolution means 4, 6 and 8 is implementing using 

a two dimensional finite impulse response filter 20. This filter 20 
has conventional pixel and line delays between taps. The input video 
data is pipelined through the filter 20 in sequence to generate the 
output. 

35 The filter coefficients applied by the filters 20 are stored 

within a coefficient store 24. The coefficient stores 24 is supplied 
with the capture time of the video data and a value of the horizontal 



vector component and vertical vector component of the pixel values 
currently at the centre position within the filter 20. 

The coefficient store 24 outputs n horizontal coefficient values 
and m vertical coefficient values to the filter 20 in dependence upon 
the horizontal vector component, the vertical vector component and the 
capture time. The values of the coefficients are determined in advance 
for particular combinations of vector component values and capture time 
using the mathematical treatment referenced in the introduction. 

The blurring transfer function may be represented by: 

H ( U ) = T sin (it (ua+vb) ) e "^ iua * vi>) 

B 7i (ua+vb) 



where : 

u = horizontal spatial frequency; 
v = vertical spatial frequency; 
T = capture time; 

a = (horizontal motion vector) * T; and 
b = (vertical motion vector) * T. 

Accordingly, the deblurring transfer function may be represented 

by: 

H D {u,v) = — i - 

H B (u,v) 

By substituting into this equation the spatial frequencies 
corresponding to the differing horizontal pixel spacings and interline 
spacings in the vertical direction, a set of filter coefficient values 
for representative values of the capture time T and the motion vector 
might be derived- Unfortunately, the value of H D (u,v) becomes infinite 
at the zero crossing that occur within H B (u,v); the first zero crossing 
occurs at | ua+vb | = 1. 

One way of dealing with this to provide approximate solutions for 
the filter coefficients without the zero crossing problem is to only 
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consider up to the first zero crossing of H B (u,v) and then apply a 
window function. An example of a window function W(u,v) that may be 
used is a raised cosine: 

W(u, v) = 1/2 (1 + cos (7t (ua+vi) ) ) .... for \ua+vb\<l 



W(u, v) = 0 .... elsewhere 



Thus, the applied non-separable deblur function becomes: 



D H B (u,v) 



An alternative is to use a least-mean-square (Wiener) filter as 
described in the mathematical treatment referenced in the introduction. 
10 The Wiener filter given by equation 5»5~10 in this reference (where K 

is a constant) is: 

1 x \H B {u.v)\* 
H B {u.v) \H B (U,V) | 2 + K 



H' D (u,v) - 



0/ 
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CLAIMS 

1. Apparatus for processing an array of image data captured during 
a capture time such that moving images are blurred, said apparatus 
5 comprising: 

means for detecting an array of motion vectors associated with 
corresponding portions of an array of image data, each of said motion 
vectors representing image motion of a corresponding one of said 
portions between temporally spaced arrays of image data; and 
10 means for deconvolving a blur function from each of said portions 

in dependence upon said capture time and a corresponding motion vector 
for that portion detected by said means for detecting to yield an array 
of reduced blur image data. 

15 2. Apparatus as claimed in claim 1, wherein said portions comprise 

arrays of pixel values. 

3. Apparatus as claimed in claims 2, wherein said means for 
detecting an array of motion vectors detects a plurality of candidate 

20 motion vectors for each of said arrays of pixel values and a means for 

selecting selects a motion vector to be associated with each pixel 
value from among said candidate motion vectors. 

4. Apparatus as claimed in claim 3» wherein said means for 
25 deconvolving performs a deconvolution upon each pixel value for each of 

said candidate motion vectors to yield a plurality of reduced blur 
pixel values and a multiplexer controlled by said means for selecting 
switches one of said candidate reduced blur image pixel values 
corresponding to said selected motion vector to form part of said array 
30 of reduced blur image data. 

5. Apparatus as claimed in any one of claims 2, 3 and 4, comprising 
means for detecting boundaries between areas within said array of image 
data sharing a substantially common motion vector, sharing of a 

35 substantially common motion vector within such an area being indicative 

of said area representing a single object. 
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6. Apparatus as claimed in claim 5. comprising a mixer responsive to 
said means For detecting boundaries for mixing together deblurred image 
data from said means for deconvolving and unprocessed image data such 
that at boundaries between detected areas substantially wholly 
unprocessed image data is output from said mixer with an increasing 
proportion of deblurred image data being output from said mixer upon 
moving away from said boundaries. 

7. Apparatus as claimed in any one of the preceding claims, wherein 
said means for deconvolving comprises a two dimensional finite impulse 
response filter having a set of filter coefficients selected in 
dependence upon motion vector components in orthogonal directions for 
that portion of image data being processed. 

8. Apparatus as claimed in claim 7. comprising a filter coefficient 
store for said finite impulse response filter storing sets of filter 
coefficients for a range of possible motion vector components , detected 
motion vector component triggering selection of a corresponding set of 
filter coefficients. 

9* Apparatus as claimed in claim 8, wherein said filter coefficient 
store stores a plurality of sets of filter coefficients for each 
possible motion vector, each of said plurality of sets corresponding to 
a different capture time. 

10. A method of processing an array of image data captured during a 
capture time such that moving images are blurred, said method 
comprising the steps of: 

detecting an array of motion vectors associated with 
corresponding portions of an array of image data, each of said motion 
vectors representing image motion of a corresponding one of said 
portions between temporally spaced arrays of image data; and 

deconvolving a blur function from each of said portions in 
dependence upon said capture time and a corresponding detected motion 
vector for that portion to yield an array of reduced blur image data. 

11. Apparatus for processing arrays of image data substantially as 
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hereinbefore described with reference to the accompanying drawings 

12. A method of processing arrays of image data substantially 
hereinbefore described with reference to the accompanying drawings 
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